14 research outputs found
Surgical Subtask Automation for Intraluminal Procedures using Deep Reinforcement Learning
Intraluminal procedures have opened up a new sub-field of minimally invasive surgery that use flexible instruments to navigate through complex luminal structures of the body, resulting in reduced invasiveness and improved patient benefits. One of the major challenges in this field is the accurate and precise control of the instrument inside the human body. Robotics has emerged as a promising solution to this problem. However, to achieve successful robotic intraluminal interventions, the control of the instrument needs to be automated to a large extent. The thesis first examines the state-of-the-art in intraluminal surgical robotics and identifies the key challenges in this field, which include the need for safe and effective tool manipulation, and the ability to adapt to unexpected changes in the luminal environment. To address these challenges, the thesis proposes several levels of autonomy that enable the robotic system to perform individual subtasks autonomously, while still allowing the surgeon to retain overall control of the procedure. The approach facilitates the development of specialized algorithms such as Deep Reinforcement Learning (DRL) for subtasks like navigation and tissue manipulation to produce robust surgical gestures. Additionally, the thesis proposes a safety framework that provides formal guarantees to prevent risky actions. The presented approaches are evaluated through a series of experiments using simulation and robotic platforms. The experiments demonstrate that subtask automation can improve the accuracy and efficiency of tool positioning and tissue manipulation, while also reducing the cognitive load on the surgeon. The results of this research have the potential to improve the reliability and safety of intraluminal surgical interventions, ultimately leading to better outcomes for patients and surgeons
On Simple Reactive Neural Networks for Behaviour-Based Reinforcement Learning
We present a behaviour-based reinforcement learning approach, inspired by Brook’s subsumption architecture, in which simple fully connected networks are trained as reactive behaviours. Our working assumption is that a pick and place robotic task can be simplified by leveraging domain knowledge of a robotics developer to decompose and train reactive behaviours; namely, approach, grasp, and retract. Then the robot autonomously learns how to combine reactive behaviours via an Actor-Critic architecture. We use an Actor-Critic policy to determine the activation and inhibition mechanisms of the reactive behaviours in a particular temporal sequence. We validate our approach in a simulated robot environment where the task is about picking a block and taking it to a target position while orienting the gripper from a top grasp. The latter represents an extra degree-of-freedom of which current end-to-end reinforcement learning approaches fail to generalise. Our findings suggest that robotic learning can be more effective if each behaviour is learnt in isolation and then combined them to accomplish the task. That is, our approach learns the pick and place task in 8,000 episodes, which represents a drastic reduction in the number of training episodes required by an end-to-end approach ( 95,000 episodes) and existing state-of-the-art algorithms
Intrinsic Robotic Introspection: Learning Internal States From Neuron Activations
We present an introspective framework inspired by the process of how humans
perform introspection. Our working assumption is that neural network
activations encode information, and building internal states from these
activations can improve the performance of an actor-critic model. We perform
experiments where we first train a Variational Autoencoder model to reconstruct
the activations of a feature extraction network and use the latent space to
improve the performance of an actor-critic when deciding which low-level
robotic behaviour to execute. We show that internal states reduce the number of
episodes needed by about 1300 episodes while training an actor-critic, denoting
faster convergence to get a high success value while completing a robotic task.Comment: Paper accepted at the International Conference on Development and
Learning (ICDL) 202
UnityFlexML: Training Reinforcement Learning Agents in a Simulated Surgical Environment
Sim-to-real Deep Reinforcement Learning (DRL) has shown promising in subtasks automation for surgical robotic systems, since it allows to safely perform all the trial and error attempts needed to learn the optimal control policy. However, a realistic simulation environment is essential to guarantee direct transfer of the learnt policy from the simulated to the real system. In this work, we introduce UnityFlexML, an open-source framework providing support for soft bodies simulation and state-of-the-art DRL methods. We demonstrate that a DRL agent can be successfully trained within UnityFlexML to manipulate deformable fat tissues for tumor exposure during a nephrectomy procedure. Furthermore, we show that the learned policy can be directly deployed on the da Vinci Research Kit, which is able to execute the trajectories generated by the DRL agent. The proposed framework represents an essential component for the development of autonomous robotic systems, where the interaction with the deformable anatomical environment is involved
Soft Tissue Simulation Environment to Learn Manipulation Tasks in Autonomous Robotic Surgery
Reinforcement Learning (RL) methods have demonstrated promising results for the automation of subtasks in surgical robotic systems. Since many trial and error attempts are required to learn the optimal control policy, RL agent training can be performed in simulation and the learned behavior can be then deployed in real environments. In this work, we introduce an open-source simulation environment providing support for position based dynamics soft bodies simulation and state-of-the-art RL methods. We demonstrate the capabilities of the proposed framework by training an RL agent based on Proximal Policy Optimization in fat tissue manipulation for tumor exposure during a nephrectomy procedure. Leveraging on a preliminary optimization of the simulation parameters, we show that our agent is able to learn the task on a virtual replica of the anatomical environment. The learned behavior is robust to changes in the initial end-effector position. Furthermore, we show that the learned policy can be directly deployed on the da Vinci Research Kit, which is able to execute the trajectories generated by the RL agent. The proposed simulation environment represents an essential component for the development of next-generation robotic systems, where the interaction with the deformable anatomical environment is involved
Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks
Deep Reinforcement Learning (DRL) is emerging as a promising approach to
generate adaptive behaviors for robotic platforms. However, a major drawback of
using DRL is the data-hungry training regime that requires millions of trial
and error attempts, which is impractical when running experiments on robotic
systems. Learning from Demonstrations (LfD) has been introduced to solve this
issue by cloning the behavior of expert demonstrations. However, LfD requires a
large number of demonstrations that are difficult to be acquired since
dedicated complex setups are required. To overcome these limitations, we
propose a multi-subtask reinforcement learning methodology where complex pick
and place tasks can be decomposed into low-level subtasks. These subtasks are
parametrized as expert networks and learned via DRL methods. Trained subtasks
are then combined by a high-level choreographer to accomplish the intended pick
and place task considering different initial configurations. As a testbed, we
use a pick and place robotic simulator to demonstrate our methodology and show
that our method outperforms a benchmark methodology based on LfD in terms of
sample-efficiency. We transfer the learned policy to the real robotic system
and demonstrate robust grasping using various geometric-shaped objects.Comment: This work has been accepted to the IEEE International Conference on
Advanced Robotics (ICAR) 202
Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation
The field of robotic Flexible Endoscopes (FEs) has progressed significantly,
offering a promising solution to reduce patient discomfort. However, the
limited autonomy of most robotic FEs results in non-intuitive and challenging
manoeuvres, constraining their application in clinical settings. While previous
studies have employed lumen tracking for autonomous navigation, they fail to
adapt to the presence of obstructions and sharp turns when the endoscope faces
the colon wall. In this work, we propose a Deep Reinforcement Learning
(DRL)-based navigation strategy that eliminates the need for lumen tracking.
However, the use of DRL methods poses safety risks as they do not account for
potential hazards associated with the actions taken. To ensure safety, we
exploit a Constrained Reinforcement Learning (CRL) method to restrict the
policy in a predefined safety regime. Moreover, we present a model selection
strategy that utilises Formal Verification (FV) to choose a policy that is
entirely safe before deployment. We validate our approach in a virtual
colonoscopy environment and report that out of the 300 trained policies, we
could identify three policies that are entirely safe. Our work demonstrates
that CRL, combined with model selection through FV, can improve the robustness
and safety of robotic behaviour in surgical applications.Comment: Accepted in the IEEE International Conference on Intelligent Robots
and Systems (IROS), 2023. [Corsi, Marzari and Pore contributed equally
Framework for soft tissue manipulation and control using Deep Reinforcement Learning
We introduce a soft-tissue simulation framework that replicates the preliminary steps of partial nephrectomy procedure. Furthermore, we show that an end-to-end reinforcement learning algorithm can be trained in the simulation without any user demonstration to accomplish a tissue manipulation task. To the best of our knowledge, this is one of the first attempts of using DRL agents to manipulate soft tissues for autonomous surgical action execution
Learning from Demonstrations for Autonomous Soft-tissue Retraction
The current research focus in Robot-Assisted Minimally Invasive Surgery (RAMIS) is directed towards increasing the level of robot autonomy, to place surgeons in a supervisory position. Although Learning from Demonstrations (LfD) approaches are among the preferred ways for an autonomous surgical system to learn expert gestures, they require a high number of demonstrations and show poor generalization to the variable conditions of the surgical environment. In this work, we propose an LfD methodology based on Generative Adversarial Imitation Learning (GAIL) that is built on a Deep Reinforcement Learning (DRL) setting. GAIL combines generative adversarial networks to learn the distribution of expert trajectories with a DRL setting to ensure generalisation of trajectories providing human-like behaviour. We consider automation of tissue retraction, a common RAMIS task that involves soft tissues manipulation to expose a region of interest. In our proposed methodology, a small set of expert trajectories can be acquired through the da Vinci Research Kit (dVRK) and used to train the proposed LfD method inside a simulated environment. Results indicate that our methodology can accomplish the tissue retraction task with human-like behaviour while being more sample-efficient than the baseline DRL method. Towards the end, we show that the learnt policies can be successfully transferred to the real robotic platform and deployed for soft tissue retraction on a synthetic phantom
Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks
Deep Reinforcement Learning (DRL) is emerging as a promising approach to generate adaptive behaviors for robotic platforms. However, a major drawback of using DRL is the data-hungry training regime that requires millions of trial and error attempts, which is impractical when running experiments on robotic systems. Learning from Demonstrations (LfD) has been introduced to solve this issue by cloning the behavior of expert demonstrations. However, LfD requires a large number of demonstrations that are difficult to be acquired since dedicated complex setups are required. To overcome these limitations, we propose a multi-subtask reinforcement learning methodology where complex pick and place tasks can be decomposed into low-level subtasks. These subtasks are parametrized as expert networks and learned via DRL methods. Trained subtasks are then combined by a high-level choreographer to accomplish the intended pick and place task considering different initial configurations. As a testbed, we use a pick and place robotic simulator to demonstrate our methodology and show that our method outperforms a benchmark methodology based on LfD in terms of sample-efficiency. We transfer the learned policy to the real robotic system and demonstrate robust grasping using various geometric-shaped objects